This section is intended to introduce the reader to various aspects of the art that may be related to various aspects of the present invention. The following discussion is intended to provide information to facilitate a better understanding of the present invention. Accordingly, it should be understood that statements in the following discussion are to be read in this light, and not as admissions of prior art.
In prior art, Rosenberg et al teach how to capture a time-varying two dimensional array of pressure upon a surface in a way that properly interpolates sensed pressure at points between individual sensing elements. This is an improvement over previous methods, such as that of TekScan, which do not interpolate between sensing elements, and therefore must use a very finely spaced two dimensional sensing element array to approximate capture of the continuous pressure image.
Moreover, Gesture sensing based only on range imaging cameras can be very powerful, since it can track entire hand or foot movements, maintain consistent identity over time of each hand of each user, and in some cases provide unambiguous finger and toe identity (depending on distance of camera to surface and hand or foot position). This stands in marked contrast to purely surface-based Touch Devices, such as those based on variable resistance or capacitance, which provide little or no information about finger and hand position or toe and foot position in the space above the surface. Yet range imaging camera suffers from several deficiencies:
(1) Frame rate (30 fps for the Kinect) is too slow to properly sample the movement of a finger pressing down and releasing a key. By way of comparison, the standard sampling rate for USB keyboards is 125 Hz (more than four times video rate). This higher sampling rate is needed for unambiguous detection and disambiguation of multiple overlapping typed keystrokes.
(2) It is impossible to determine from a range image alone how much pressure is being applied to a surface, thereby rendering range imaging cameras inadequate for subtle movement of virtual objects on a display, rapid and accurate control of 3D computer game characters, musical instrument emulation, simulated surgery, simulated painting/sculpting, gait monitoring, dance, monitoring stance for purposes of physical therapy, and other applications that benefit from a significant measure of isometric control.
It is therefore also impossible to determine from a 3D image gestures based on movements and variations in pressure on the underside of fingers or hands or feet or toes. For example, if a user shifts weight between different fingers, or between fingers and different parts of the palm, or between the foot heel, metatarsal or toes, these changes will be undetectable to a range imaging camera.
The decade of 2001-2011 has seen the gradual development of LCD displays that contain an optically sensitive element in each pixel (variously developed by Sharp, Toshiba and Matsushita). This approach enables the sensing of both touch and hovering. However, the optically sensitive pixel approach suffers from a number of deficiencies as compared to the present touch-range fusion apparatus approach: (1) The cost per unit area is intrinsically far higher than the cost per unit area of the approach here; (2) Such sensors cannot be seamlessly tiled to arbitrarily large form factors; (3) variations in the pressure of a detected touch 111 can be determined only with very low fidelity (via changes in fingertip contact shape); (4) hand shape can only be detected within a relatively small distance above the display. This makes it impossible to maintain a persistent model of hand and finger identity or to recognize many hand gestures. In addition, it is not practical to use such technologies for foot sensing, since the added cost to manufacture such sensors so that they possess sufficient physical robustness to withstand the weight of a human body would add prohibitively to their cost.